Device for extracting depth information and method thereof

ABSTRACT

A device for extracting depth information according to one embodiment of the present invention comprises: a light outputting unit for outputting IR (InfraRed) light; a light inputting unit for inputting light reflected from an object after outputting from the light outputting unit; a light adjusting unit for adjusting the angle of the light so as to radiate the light into a first area including the object, and then for adjusting the angle of the light so as to radiate the light into a second area; and a controlling unit for estimating the motion of the object by using at least one of the lights between the light inputted to the first area and the light inputted to the second area.

TECHNICAL FIELD

The present disclosure relates to extracting depth information, and moreparticularly, to a device for extracting depth information using atime-of-flight (TOF) method, and a method thereof.

BACKGROUND ART

A technology of acquiring a three-dimensional image using a capturingdevice is developing. Depth information (depth map) is required foracquiring a three-dimensional image. Depth information is informationthat indicates a spatial distance and shows perspective information of apoint with respect to another point in a two-dimensional image.

A method in which infrared (IR) structured light is projected to anobject and light reflected from the object is interpreted to extractdepth information is one of the methods of acquiring depth information.According to the method using the IR structured light, there is aproblem in that it is difficult to obtain a desired level of depthresolution for a moving object.

A time-of-flight (TOF) method is gaining attention as a technology forsubstituting the method using IR structured light. According to the TOFmethod, a distance from an object is calculated by measuring time offlight, i.e., the time taken for emitted light to be reflected.

Generally, a camera according to the TOF method adjusts the angle oflight to scan a front surface of an object. The TOF camera has problemsof having low optical efficiency and a large number of operations.

DISCLOSURE Technical Problem

The present disclosure is directed to providing a device and method forextracting depth information, in which a TOF method is used to extractdepth information.

Technical Solution

According to an embodiment of the present disclosure, a device forextracting depth information includes a light outputting unit thatoutputs infrared (IR) light, a light inputting unit that is input withlight output from the light outputting unit and then reflected from anobject, a light adjusting unit that adjusts an angle of the light suchthat a first region including an object is irradiated with the light andthen adjusts the angle of the light such that a second region, which isa portion of the first region, is irradiated with the light, and acontrolling unit that estimates motion in the second region using beamsof light sequentially input into the second region over time.

The first region may be an entire region including the object, and thesecond region may be extracted from the first region and may be apartial region including a predetermined region of the object.

The controlling unit may calculate a time of flight taken for lightoutput from the light outputting unit to be input into the lightinputting unit after being reflected from the second region that isirradiated with the light.

The controlling unit may estimate a motion in the second region using afirst time of flight calculated using light input at a first time point,a second time of flight calculated using light input before the firsttime point, and a third time of flight calculated using light inputafter a second time point.

The controlling unit may estimate a motion in the second region usingrelative differences between the first time of flight, the second timeof flight, and the third time of flight and then compensate for theestimated motion using an interpolation technique.

The light inputting unit may include a plurality of pixels eachincluding a first reception unit and a second reception unit, and thecontrolling unit may calculate the time of flight using a difference inamounts of light input into the first reception unit and the secondreception unit.

The light adjusting unit may include a microelectromechanical system(MEMS) and a MEMS controlling unit.

The controlling unit may include a timing controlling unit that controlstime points of the light outputting unit, the light adjusting unit, andthe light inputting unit, a conversion unit that converts an electricalsignal input through the light inputting unit into a digital signal, anda signal processing unit that calculates times of flight of beams oflight sequentially input into the second region over time, estimates amotion in the second region, and extracts depth information of thesecond region.

According to an embodiment of the present disclosure, a method forextracting depth information includes irradiating a first regionincluding an object with infrared (IR) light, irradiating a secondregion which is a portion of the first region with IR light, andestimating motion in the second region using beams of light sequentiallyinput into the second region over time.

The estimating may include calculating a first time of flight withrespect to light input at a first time point, a second time of flightwith respect to light input before the first time point, and a thirdtime of flight with respect to light input after a second time point andestimating motion in the second region using relative differencesbetween the first time of flight, the second time of flight, and thethird time of flight.

According to another embodiment of the present disclosure, a device forextracting depth information includes a light source that irradiates anobject with light, a holographic element arranged between the lightsource and the object to adjust an irradiation region of the lightsource, an actuator that drives the holographic element so that theradiation region is different in first and second frame periods, alight-receiving lens that receives reflected light reflected by theobject, a sensor unit that receives the reflected light through thelight-receiving lens and synchronizes with each of the first and secondframe periods to output first and second image signals, a signalprocessing unit that processes the first and second image signals togenerate first and second frames, and a deinterlacer that merges thefirst and second frames to each other to generate a depth image.

According to yet another embodiment of the present disclosure, a devicefor extracting depth information includes first and second light sourcesthat are arranged to be spaced apart from each other and irradiate anobject with light, first and second holographic elements arrangedbetween the first and second light sources and the object to adjustirradiation regions of the first and second light sources so that theirradiation regions of the first and second light sources are differentfrom each other, a controlling unit that controls lighting of the firstand second light sources so that the first light source emits light inthe first frame period and the second light source emits light in thesecond frame period, a light-receiving lens that receives reflectedlight reflected by the object, a sensor unit that receives the reflectedlight through the light-receiving lens and synchronizes with each of thefirst and second frame periods to output first and second image signals,a signal processing unit that processes the first and second imagesignals to generate first and second frames, and a deinterlacer thatmerges the first and second frames to each other to generate a depthimage.

Advantageous Effects

According to an embodiment of the present disclosure, a device forextracting depth information with a small number of operations andexcellent depth resolution can be obtained. Accordingly, electricityconsumed by the device for extracting depth information can be reduced,and a distance from an object can be precisely extracted.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a depth information extracting systemaccording to an embodiment of the present disclosure.

FIG. 2 illustrates a structure of a light inputting unit of a depthinformation extracting device according to an embodiment of the presentdisclosure.

FIG. 3 illustrates the principle of extracting depth information of adepth information extracting device according to an embodiment of thepresent disclosure.

FIG. 4 illustrates an entire region and a partial region sequentiallycaptured over time.

FIGS. 5 and 6 are flowcharts illustrating a depth information extractingmethod of a depth information extracting device according to anembodiment of the present disclosure.

FIG. 7 is a block diagram schematically illustrating a depth informationextracting device according to another embodiment of the presentdisclosure.

FIG. 8 is a view for describing an example in which an emission angle ofthe depth information extracting device according to another embodimentof the present disclosure is controlled by a holographic element.

FIG. 9 is a view for describing a method in which the depth informationextracting device according to another embodiment of the presentdisclosure generates frames by synchronizing with frame periods.

FIG. 10 is a view illustrating the depth information extracting deviceaccording to another embodiment of the present disclosure generating adepth image by merging odd-numbered frames and even-numbered frames.

FIG. 11 is a flowchart illustrating a depth information extractingmethod of the depth information extracting device according to anotherembodiment of the present disclosure.

FIG. 12 is a block diagram schematically illustrating a depthinformation extracting device according to yet another embodiment of thepresent disclosure.

FIG. 13 is a view for describing a method in which the depth informationextracting device according to yet another embodiment of the presentdisclosure extracts depth information.

FIG. 14 is a flowchart illustrating a method of extracting depthinformation by the depth information extracting device according to yetanother embodiment of the present disclosure.

MODES OF THE INVENTION

Since various modifications may be made to the present disclosure andthe present disclosure may have various embodiments, particularembodiments will be illustrated in the drawings and described. However,this does not limit the present disclosure to the particularembodiments, and all modifications, equivalents, and substitutesincluded within the spirit and scope of the present disclosure should beconstrued as belonging to the present disclosure.

Terms including ordinals such as first and second may be used todescribe various elements, but the elements are not limited by theterms. The terms are only used for the purpose of distinguishing oneelement from another element. For example, a second element may bereferred to as a first element while not departing from the scope of thepresent disclosure, and likewise, a first element may also be referredto as a second element. The term and/or includes a combination of aplurality of related described items or any one item among the pluralityof related described items.

When it is mentioned that a certain element is “connected” or “linked”to another element, although the certain element may be directlyconnected or linked to the other element, it should be understood thatanother element may exist therebetween. On the other hand, when it ismentioned that a certain element is “directly connected” or “directlylinked” to another element, it should be understood that other elementsdo not exist therebetween.

Terms used in the application are merely used to describe particularembodiments and are not intended to limit the present disclosure. Asingular expression includes a plural expression unless the contextclearly indicates otherwise. In the application, terms such as “include”or “have” should be understood as designating that features, number,steps, operations, elements, parts, or combinations thereof exist andnot as precluding the existence of or the possibility of adding one ormore other features, numbers, steps, operations, elements, parts, orcombinations thereof in advance.

Unless otherwise defined, all terms including technical or scientificterms used herein have the same meaning as commonly understood by one ofordinary skill in the art to which the disclosure pertains. Terms, suchas those defined in commonly used dictionaries, should be construed ashaving a meaning that is consistent with their meaning in the context ofthe relevant art and are not to be construed in an idealized or overlyformal sense unless expressly so defined herein.

Hereinafter, an embodiment will be described in detail with reference tothe accompanying drawings while like reference numerals will be given tothe same or corresponding elements regardless of signs in the drawingsand overlapping descriptions thereof will be omitted.

According to an embodiment of the present disclosure, an angle ofradiated light is adjusted to improve depth resolution.

According to an embodiment of the present disclosure, depth informationof a partial region is extracted by irradiating an entire regionincluding an object with light, extracting a partial region from theentire region, and repeatedly irradiating the extracted partial regionwith light.

FIG. 1 is a block diagram of a depth information extracting systemaccording to an embodiment of the present disclosure, FIG. 2 illustratesa structure of a light inputting unit of a depth information extractingdevice according to an embodiment of the present disclosure, and FIG. 3illustrates the principle of extracting depth information of a depthinformation extracting device according to an embodiment of the presentdisclosure.

Referring to FIG. 1, a depth information extracting system includes adepth information extracting device 100 and a personal computer (PC)200. The depth information extracting device may be a three-dimensionalstereoscopic camera or a portion thereof.

The depth information extracting device 100 includes a light outputtingunit 110, a light adjusting unit 120, a light inputting unit 130, and acontrolling unit 140.

The light outputting unit 110 outputs infrared (IR) light. The IR lightmay be, for example, light having a wavelength band that is 800 nm orhigher. The light outputting unit 110 includes a light source 112 and alight conversion unit 114. The light source may include at least onelaser diode (LD) or light emitting diode (LED) that projects infraredlight. Also, the light conversion unit 114 may modulate light outputfrom the light source 112. The light conversion unit 114 may, forexample, perform pulse modulation or phase modulation of the lightoutput from the light source 112. Accordingly, the light outputting unit110 may output light while causing the light source to flicker at everypredetermined interval.

The light adjusting unit 120 adjusts an angle of light so that a regionincluding an object is irradiated with light. For this, the lightadjusting unit 120 may include a microelectromechanical system (MEMS)actuator 122 and a MEMS controlling unit 124.

The light adjusting unit 120 may adjust the angle of light so that anentire region including an object is irradiated with light. For example,as in FIG. 4(A), the light adjusting unit 120 may adjust the angle oflight so that an entire region including a person is irradiated with thelight. Accordingly, light output from the light outputting unit 110 mayscan the entire region in units of pixels or lines. Also, the lightadjusting unit 120 may also adjust the angle of light so that a partialregion, which is a portion of the entire region, is irradiated with thelight. For example, as illustrated in FIG. 4(B), the light adjustingunit 120 may adjust the angle of light so that a partial regionincluding hands of the entire region is irradiated with the light.Accordingly, the light output from the light outputting unit 110 mayscan only the partial region in units of pixels or lines.

Meanwhile, the light inputting unit 130 is input with light output fromthe light outputting unit 110 and reflected by an object. The lightinputting unit 130 may convert the input light into an electricalsignal. The light inputting unit 130 may be an image sensor including aphoto diode (PD) or a complementary metal-oxide semiconductor (CMOS). Asin FIG. 2, the light inputting unit 130 may include a plurality ofpixels 132 arranged. Each pixel may include an in-phase reception unit132-1 and an out-phase reception unit 132-2.

The controlling unit 140 controls an overall operation of the depthinformation extracting device 100 and extracts depth information. Thecontrolling unit 140 may be implemented with a controller chip. Thecontrolling unit 140 may include a timing controlling unit 142, aconversion unit 144, a signal processing unit 146, and an interfacecontroller 148. The timing controlling unit 142 controls time points ofthe light outputting unit 110, the light adjusting unit 120, and thelight inputting unit 130. For example, the timing controlling unit 142may control the flickering cycle of the light outputting unit 110. Theconversion unit 144 may convert an electrical signal input through thelight inputting unit 130 into a digital signal.

In addition, the signal processing unit 146 calculates times of flightof beams of light sequentially input into the partial region over time,estimates motion in the partial region, and extracts depth informationof the partial region. Here, the time of flight of the light may becalculated using a difference between amounts of light input into thein-phase reception unit 132-1 and the out-phase reception unit 132-2.That is, as in FIG. 3, the in-phase reception unit 132-1 may beactivated while a light source is turned on, and the out-phase receptionunit 132-2 may be activated while the light source is turned off. Inthis way, when the in-phase reception unit 132-1 and the out-phasereception unit 132-2 are activated with a time difference, a differenceoccurs in the time of flight of light, i.e., an amount of light receivedaccording to a distance from an object. For example, when the object isright in front of the depth information extracting device (i.e., whendistance=0), the time taken for light output from the light outputtingunit 110 to be reflected is 0 such that a flickering cycle of a lightsource becomes a light reception cycle without change. Accordingly, onlythe in-phase reception unit 132-1 receives light, and the out-phasereception unit 132-2 does not receive light. In another example, whenthe object is spaced a predetermined distance away from the depthinformation extracting device, a time is taken for light output from thelight outputting unit 110 to be reflected such that the flickering cycleof the light source is different from the light reception cycle.Accordingly, a difference occurs between amounts of light received bythe in-phase reception unit 132-1 and the out-phase reception unit132-2.

The interface controller 148 controls an interface with middleware suchas the PC 200. For example, the interface controller 148 may transmitinformation on light input through the light inputting unit 130 afterbeing radiated to an entire region to middleware such as the PC 200.Also, the interface controller 148 may receive information on a partialregion extracted by the middleware such as the PC 200 from themiddleware such as the PC 200 and then transmit the information to thelight adjusting unit 120 and the like.

FIGS. 5 and 6 are flowcharts illustrating a depth information extractingmethod of a depth information extracting device according to anembodiment of the present disclosure. Descriptions of contentsoverlapping with those of FIGS. 1 to 3 will be omitted.

Referring to FIG. 5, the light outputting unit 110 of the depthinformation extracting device 100 outputs IR light (S500), and an entireregion including an object is irradiated with the output light byadjustment by the light adjusting unit 120 (S502).

In addition, light reflected from the object is input through the lightinputting unit 130 (S504), and the controlling unit 140 converts ananalog signal received from the light inputting unit 130 into a digitalsignal (S506) and then transmits the digital signal to the PC 200, forexample (S508).

The PC 200 uses the signal received from the depth informationextracting device 100 and extracts a partial region of the entire region(S510). The partial region may be a region including an interest targetrequired for implementing an application. For example, when theapplication is changing a television (TV) channel according to a gestureof a finger, the partial region may include only fingers when the entireregion includes the whole body of a person.

The PC 200 transmits information on the extracted partial region to thedepth information extracting device 100 (S512).

Meanwhile, the light outputting unit 110 of the depth informationextracting device 100 outputs IR light (S514), and the output light isradiated only to the partial region extracted from the entire region byadjustment by the light adjusting unit 120 (S516).

In addition, light reflected from an object unit is input through thelight inputting unit 130 (S518), the controlling unit 140 converts theanalog signal received from the light inputting unit 130 into a digitalsignal (S520), and signal processing is performed to extract depthinformation of the partial region (S522).

According to an embodiment of the present disclosure, the depthinformation extracting device 100 may scan a partial region severaltimes during the time over which an entire region can be scanned once.Accordingly, Step S514 to Step S520 are repeated several times, andprecision of depth information can be improved using the result repeatedfor multiple times in Step S522. This will be described in detail withreference to FIG. 6.

Referring to FIG. 6, with respect to a partial region, the controllingunit 140 of the depth information extracting device calculates a time offlight of light input through the light inputting unit 130 at a time T1(S600), calculates a time of flight of light input through the lightinputting unit 130 at a time T2 (S602), and calculates a time of flightof light input through the light inputting unit 130 at a time T3 (S604).Here, T1, T2, and T3 may have the same time intervals, and Step S514 toStep S520 in FIG. 5 may be repeated at each of T1, T2, and T3. Asdescribed with reference to FIG. 3, the time of flight of light may becalculated using a difference in the amount of light between an in-phasereception unit and an out-phase reception unit.

In addition, the controlling unit 140 estimates a motion in the partialregion based on the time of flights at the time T1 to the time T3 (S606)and extracts depth information according to a result of the motionestimation (S608). Here, processes of the motion estimation and thedepth information extraction may be performed according to a superresolution (SR) algorithm. That is, as illustrated in FIG. 4(B), aftercapturing the partial region from T1 to Tn, time of flight at each timemay be calculated to estimate relative motion over time. In addition,the estimated motions may be compensated using the interpolationtechnique and restored, and noise may be reduced therefrom.

In this way, when the partial region is detected from the entire regionand depth information on the partial region is extracted, complexity ofcalculation may be decreased. In addition, because motion information isestimated with respect to the partial region using relative differencesin information sequentially input over time, high depth resolution canbe obtained. Also, because the partial region may be scanned severaltimes during the time over which the entire region is scanned once, anamount of time for and complexity of calculation of depth informationextraction can be decreased.

According to another embodiment of the present disclosure, depthinformation is extracted by repeating a process of irradiating a partialregion including an object with light and adjusting the angle of thelight so that the regions are irradiated with light.

FIG. 7 is a block diagram schematically illustrating a depth informationextracting device according to another embodiment of the presentdisclosure. FIG. 8 is a view for describing an example in which theemission angle of the depth information extracting device according toanother embodiment of the present disclosure is controlled by aholographic element. FIG. 9 is a view for describing a method in whichthe depth information extracting device according to another embodimentof the present disclosure generates frames by synchronizing with frameperiods. FIG. 10 is a view illustrating the depth information extractingdevice according to another embodiment of the present disclosuregenerating a depth image by merging odd-numbered frames andeven-numbered frames. Descriptions of contents overlapping with those ofFIGS. 1 to 6 will be omitted.

Referring to FIG. 7, a depth information extracting device according toanother embodiment of the present disclosure may include a depth sensormodule 10, a signal processing unit 31, and a deinterlacer 32. Theelements illustrated in FIG. 7 are not essential, and thus a depthinformation extracting device according to an embodiment of the presentdisclosure may include more or less elements.

The depth sensor module 10 is a time of flight (TOF) sensor module andmay include a light emitting unit and a light receiving unit.

The light emitting unit of the depth sensor module 10 may include alight source 11, a lens 12, a holographic element 13, an actuator 14,and a controlling unit 15. Here, the light source 11 may be usedtogether with a light outputting unit, and the holographic element 13and the actuator 14 may be used together with a light adjusting unit.

The light source 11 includes a light emitting element and serves todrive the light emitting element to irradiate an object OB with lighthaving a predetermined phase. The light source 11 may synchronize with apreset frame period and operate to repeat flickering.

Although the light emitting element included in the light source 11 mayinclude a laser, a laser diode, etc., other types of light sources mayalso be used. Light radiated from laser or a laser diode has relativelybetter directionality than light radiated from a light emitting diode(LED). Consequently, when a laser or a laser diode is used as the lightsource 11, the emission angle and radiation region may be easilyadjusted.

The wavelength of light radiated from the light source 11 may beincluded in an IR band but may also be included in other wavelengthbands.

The lens 12 is arranged on an optical path of the light radiated fromthe light source 11 and serves to convert light radiated from the lightsource 11, which is a point light source, into a surface light source. Alaser has excellent directionality but has a relatively small emissionangle. Consequently, the lens 12 may be used to uniformly irradiate thewhole object OB with light to acquire a depth image.

The holographic element 13 is arranged between the light source 11 andthe object OB or between the lens 12 and the object OB and serves tocontrol the emission angle and irradiation region of the light radiatedfrom the light source 11.

(A) of FIG. 8 illustrates a case in which the object OB is irradiatedwith light radiated from a light source (LED) without the holographicelement 13, and (B) of FIG. 8 illustrates a case in which the object OBis irradiated with the light radiated from the light source (LED) afterthe emission angle thereof is adjusted by the holographic element 13.

Referring to (A) of FIG. 8, regions a1 and a2, other than a region of asensor on which an image is projected are also irradiated with lightfrom the LED, thus decreasing optical efficiency.

On the other hand, referring to (B) of FIG. 8, the emission angle of thelight radiated from the LED is adjusted by the holographic element 13 sothat the light does not deviate much from the region of the sensor onwhich an image is projected (viewing angle), thus minimizing wastedlight and improving optical efficiency.

Meanwhile, since the lens 12 is not essential, the lens 12 may also beomitted when uniformity of illumination is provided by the holographicelement 13.

Again, referring to FIG. 7, the holographic element 13 may bemanufactured using a computer generated hologram (CGH) method. Ahologram is an interference pattern generated from a signal wavecontaining information scattered from an object and a coherent referencewave, and the CGH method is a method of mathematically calculating andgenerating the interference pattern using a computer.

The holographic element 13 may be formed by recording an interferencepattern calculated by a computer in a recording medium.

The recording medium in which an interference pattern is recorded in theholographic element 13 may include a substrate formed of aphotosensitive material. A photopolymer, an ultraviolet (UV)photopolymer, a photoresist, a silver halide emulsion, dichromatedgelatin, a photographic emulsion, photothermoplastic, photorefractivematerial, etc. may be used as the photosensitive material.

The holographic element 13 may be a volume holographic element or asurface holographic element.

A volume holographic element is a holographic optical element in whichan interference pattern generated in a space due to interference betweena signal wave and a reference wave is three-dimensionally recorded in arecording medium. On the other hand, a surface holographic element is aholographic optical element in which an interference pattern generatedin a space due to interference between a signal wave and a referencewave is recorded on a surface of a recording medium.

When the holographic element 13 is a surface holographic element, aninterference pattern may be formed on an exiting surface through whichlight incident from the light source 11 exits.

The holographic element 13 is connected to the actuator 14, and aposition, tilting, etc. of the holographic element may be adjusted bythe actuator 14.

The actuator 14 is a driving device that drives the holographic element13 and may adjust the position, the tilting, etc. of the holographicelement 13 with respect to an optic axis of the light source 11.

When the position, the tilting, etc. of the holographic element 13 ischanged, an irradiation region, in which the object OB is irradiatedwith light that has passed through the holographic element 13, shifts,causing a region in which light reflected by the object OB is projectedonto each cell of a sensor unit 22 to also shift.

The actuator 14 may drive the holographic element 13 based on a controlsignal of the controlling unit 15 so that the irradiation region of theholographic element 13 is switched. That is, the actuator 14 may drivethe holographic element 13 so that an irradiation region of theholographic element 13 in an odd-numbered frame period is different froman irradiation region of the holographic element 13 in an even-numberedframe period.

The irradiation region of the holographic element 13 in an odd-numberedframe period may be a region that has moved upward or downward by asmuch as a region matching one cell of the sensor unit 22 from theradiation region of the holographic element 13 in an even-numbered frameperiod. That is, the radiation region of the holographic element 13 maybe adjusted in every frame period so that a position at which an imageis projected onto a cell of the sensor unit 22 in an odd-numbered frameperiod is different from a position at which an image is projected ontoa cell of the sensor unit 22 in an even-numbered frame period by onecell. Accordingly, even when light is reflected from the same point, aposition thereof at which an image is projected onto a cell of thesensor unit 22 in an odd-numbered frame period and a position thereof atwhich an image is projected onto a cell of the sensor unit 22 in aneven-numbered frame period may be different by one cell.

The actuator 14 may be a voice coil moto (VCM) actuator or amicroelectromechanical (MEMS) actuator.

The controlling unit 15 may control the light source 11 to be turned onor off based on a preset frame period.

For example, as in FIG. 9, the controlling unit 15 may periodically turnthe light source 11 on or off so that the light source 11 flickers atevery frame period. That is, the controlling unit 15 may performoperations of turning on the light source 11 to irradiate with lightwhen frame periods T1 and T2 begin and turning off the light source 11after a predetermined amount of time passes to control irradiation withlight to be stopped until the end of the frame periods T1 and T2 everyframe periods T1 and T2.

The controlling unit 15 may control the actuator 14 so that theradiation region of the holographic element 13 is switched at everyframe period.

The light receiving unit of the depth sensor module 10 may include alight receiving lens 21 and the sensor unit 22. In this specification,the light receiving unit may be used together with the light inputtingunit.

The light receiving lens 21 is arranged between the object OB and thesensor unit 22 and serves to receive reflected light reflected from theobject OB and make the received reflected light be incident on thesensor unit 22.

The sensor unit 22 receives the reflected light reflected from theobject OB, converts the received reflected light into an electricalsignal, and outputs an image signal.

The sensor unit 22 may be formed of a plurality of cells, and each cellmay correspond to each pixel forming a frame. The sensor unit 22 mayreceive light in units of cells and output an image signal in units ofcells.

Each of the cells forming the sensor unit 22 may include an element thatconverts light into an electrical signal. Although not limited thereto,each of the cells may include a metal-oxide semiconductor (MOS), acharge coupled device (CCD), etc.

The sensor unit 22 may synchronize with a preset frame period to receivereflected light and convert the received reflected light into an imagesignal to be output.

Meanwhile, as the radiation region is switched at every frame period bythe holographic element 13, a position at which an image is projected onthe sensor unit 22 may also be switched every frame period.

The holographic element 13 irradiates, with light, different regions inan odd-numbered frame period and an even-numbered frame period by theactuator 14. Accordingly, even when light is reflected from the sameposition of the object OB, a position thereof at which an image isprojected on the sensor unit 22 in an odd-numbered frame period and aposition thereof at which an image is projected on the sensor unit 22 inan even-numbered frame period may be different.

An image projected on the sensor unit 22 in an odd-numbered frame periodmay be projected by moving upward or downward by as much as one cellfrom an image projected on the sensor unit 22 in an even-numbered frameperiod.

The signal processing unit 31 performs signal processing such assampling with respect to an image signal output from the sensor unit 22to generate a frame. Here, the frame may include a plurality of pixels,and a pixel value of each of the pixels may be acquired from an imagesignal corresponding to each of the pixels.

The signal processing unit 31 may output an image signal output from thesensor unit 22 corresponding to a frame period by converting the imagesignal into a frame.

For example, as in FIG. 9, the signal processing unit 31 may outputimage signals by converting the image signals into frames 5 a and 5 b atevery frame periods T1 and T2 and synchronize with each of anodd-numbered frame period T1 and an even-numbered frame period T2 tooutput an odd-numbered frame 5 a and an even-numbered frame 5 b.

The signal processing unit 31 may also calculate depth informationcorresponding to each cell, i.e., each pixel based on a phase differencebetween light received through each cell of the sensor unit 22 and lightradiated by the light source 11. The calculated depth information isstored corresponding to each pixel forming a frame.

The deinterlacer 32 may receive each of an odd-numbered frame and aneven-numbered frame from the signal processing unit 31 and merge the twoframes to generate one depth image.

The deinterlacer 32 may generate a depth image with resolution that hasincreased twice compared to resolution of the sensor unit 22 byalternately inserting an odd-numbered frame and an even-numbered frameoutput from the signal processing unit 31 in units of lines. Forexample, as in FIG. 10, each of the odd-numbered frame and theeven-numbered frame has a resolution of 10 (line)×10 (column) pixel, anda resolution of a depth image in which the two frames are alternatelymerged in the units of lines is 20 (line)×20 (column) pixel. Verticalresolution has increased twice compared to the resolution of the sensorunit 22. The controlling unit 15, the signal processing unit 31, and thedeinterlacer 32 of FIG. 7 may correspond to the controlling unit 14 ofFIG. 1.

FIG. 11 is a flowchart illustrating a depth information extractingmethod of the depth information extracting device according to anotherembodiment of the present disclosure.

Referring to FIG. 11, the depth sensor module 10 controls the lightsource 11 to radiate light by synchronizing with an odd-numbered frameperiod (S101).

In Step S101, an irradiation region of the light radiated from the lightsource 11 may be controlled by the holographic element 13.

As the light radiated from the light source 11 is reflected from anobject and incident on the sensor unit 22, the sensor unit 22 generatesan image signal corresponding to the light in units of pixels. Inaddition, the signal processing unit 31 performs signal processing ofthe image signal output from the sensor unit 22 to generate anodd-numbered frame (S102).

When the odd-numbered frame is acquired, the depth sensor module 10controls the holographic element 13 and shifts an irradiation region inwhich an object is irradiated with light (S103).

In Step S103, the depth sensor module 10 may shift the irradiationregion by controlling the actuator 14 to adjust a slope of theholographic member 13.

When an even-numbered frame period begins, the depth sensor module 10synchronizes with the even-numbered frame period and controls the lightsource 11 to radiate light (S104).

In Step S104, an irradiation region of the light radiated from the lightsource 11 may be different from that of the even-numbered frame perioddue to the holographic element 13.

As the light radiated from the light source 11 is reflected from anobject and incident on the sensor unit 22, the sensor unit 22 generatesan image signal corresponding to the light in units of pixels. Inaddition, the signal processing unit 31 performs signal processing ofthe image signal output from the sensor unit 22 to generate aneven-numbered frame (105).

The deinterlacer 32 generates a depth image by merging an odd-numberedframe and an even-numbered frame in units of lines of pixel arrays thatform the frames (S106).

In Step S106, the deinterlacer 32 may generate a depth image bysequentially and alternately merging lines of pixels of the odd-numberedframe and the even-numbered frame.

Hereinafter, referring to FIGS. 12 to 14, a depth information extractingmethod and a depth information extracting device for performing themethod according to yet another embodiment of the present disclosurewill be described in detail.

FIG. 12 is a block diagram schematically illustrating a depthinformation extracting device according to yet another embodiment of thepresent disclosure. FIG. 13 is a view for describing a method in whichthe depth information extracting device according to yet anotherembodiment of the present disclosure extracts depth information.

Hereinafter, to avoid overlapping descriptions, detailed descriptions ofelements having the same function as in the depth information extractingdevice that has been described with reference to FIG. 7 will be omitted.

Referring to FIG. 12, a depth information extracting device according toyet another embodiment of the present disclosure may include the depthsensor module 10, the signal processing unit 31, and the deinterlacer32. The elements illustrated in FIG. 12 are not essential, and thus thedepth information extracting device according to yet another embodimentof the present disclosure may include more or less elements.

The depth sensor module 10 is a TOF sensor module and may include aplurality of light emitting units and light receiving units.

The plurality of light emitting units forming the depth sensor module 10may each include light sources 11 a and 11 b, lenses 12 a and 12 b, andholographic elements 13 a and 13 b. The plurality of light emittingunits may further include the controlling unit 15. Here, the lightsources 11 a and 11 b may be used together with a light outputting unit,and the holographic elements 13 a and 13 b may be used together with alight adjusting unit.

The plurality of light sources 11 a and 11 b may be arranged by beingspaced a predetermined interval from each other. The sensor unit 22 ofthe depth sensor module 10 may be arranged between the plurality oflight sources 11 a and 11 b.

Hereinafter, for convenience of description, each of the plurality oflight emitting units will be referred to as a first light emitting unitand a second light emitting unit, and elements forming the first lightemitting unit will be referred to as a first light source 11 a, a firstlens 12 a, and a first holographic element 13 a. In addition, elementsforming the second light emitting unit will be referred to as a secondlight source 11 b, a second lens 12 b, and a second holographic element13 b. However, the elements are not limited by terms including ordinalssuch as first and second.

The first and second light sources 11 a and 11 b are arranged to bespaced a predetermined interval from each other and serve to drive alight emitting element to irradiate the object OB with light having apredetermined phase.

The light emitting element included in the first and second lightsources 11 a and 11 b may include a laser, a laser diode, etc., butother types of light sources may also be used. A wavelength of lightradiated from the first and second light sources 11 a and 11 b may beincluded in an IR band but may also be included in other differentwavelength bands.

Flickering of the first and second light sources 11 a and 11 b may becontrolled so that the first and second light sources 11 a and 11 b emitlight in different frame periods. For example, the first light source 11a may emit light by synchronizing with an odd-numbered frame period, andthe second light source 11 b may emit light by synchronizing with aneven-numbered frame period.

The first lens 12 a is arranged on an optical path of light radiatedfrom the first light source 11 a and serves to convert the lightradiated from the first light source 11 a which is a point light sourceinto a surface light source. In addition, the second lens 12 b isarranged on an optical path of light radiated from the second lightsource 11 b and serves to convert the light radiated from the secondlight source 11 b which is a point light source into a surface lightsource.

The first holographic element 13 a is arranged between the first lightsource 11 a and the object OB or between the first lens 12 a and theobject OB and serves to control the emission angle and irradiationregion of the light radiated from the first light source 11 a. Inaddition, the second holographic element 13 b is arranged between thesecond light source 11 b and the object OB or between the second lens 12b and the object OB and serves to control the emission angle andirradiation region of the light radiated from the second light source 11b.

Meanwhile, the first and second lenses 12 a and 12 b are not essentialand thus may be omitted when uniformity of illumination is provided bythe first and second holographic elements 13 a and 13 b.

The first and second holographic elements 13 a and 13 b may bemanufactured using the CGH method.

The first and second holographic elements 13 a and 13 b are formed byrecording an interference pattern calculated by a computer in arecording medium, and photosensitive material such as a photopolymer, aUV photopolymer, a photoresist, a silver halide emulsion, dichromatedgelatin, a photographic emulsion, photothermoplastic, andphotorefractive material may be used for the recording medium in whichthe interference pattern is recorded.

The first and second holographic elements 13 a and 13 b may be a volumeholographic element or a surface holographic element.

Interference patterns recorded in the first and second holographicelements 13 a and 13 b may be formed so that different regions of theobject OB are irradiated with light radiated by the first light source11 a and the second light source 11 b.

For example, the interference patterns of the first holographic element13 a and the second holographic element 13 b may be formed so that aregion that is irradiated with light that exited from the first lightsource 11 a and a region that is irradiated with light that exited fromthe second light source 11 b is radiated are different from each otherby as much as a region matching one cell of the sensor unit 22.

The controlling unit 15 may control the first and second light sources11 a and 11 b to be turned on or off based on a preset frame period. Forexample, as in FIG. 13, the controlling unit 15 may control the firstand second light sources 11 a and 11 b to be turned on or off so thatthe first light source 11 a emits light by synchronizing with theodd-numbered frame period T1 and the second light source 11 b emitslight by synchronizing with the even-numbered frame period T2.

Accordingly, a light source with which object OB is irradiated may beswitched at every frame period, and a region that is irradiated withlight may also be switched corresponding to the switching of the lightsource by the first and second holographic elements 13 a and 13 b.

The light receiving unit of the depth sensor module 10 may include thelight receiving lens 21 and the sensor unit 22.

The light receiving lens 21 is arranged between the object OB and thesensor unit 22 and serves to receive reflected light reflected from theobject OB and make the received reflected light be incident on thesensor unit 22.

The sensor unit 22 may be formed of a plurality of cells, and each cellmay correspond to each pixel forming a frame. The sensor unit 22 mayreceive light in units of cells and output an image signal in units ofcells.

Although not limited thereto, each of the cells forming the sensor unit22 may include an MOS, a CCD, etc.

The sensor unit 22 may synchronize with a preset frame period to receivereflected light and convert the received reflected light into an imagesignal to be output.

As a light source that radiates light to the object OB is switched everyframe period, a position at which an image is projected on the sensorunit 22 may also be switched at every frame period. That is, due to theswitching of the light source, a region in an odd-numbered frame periodirradiated with light and a region in an even-numbered frame periodirradiated with light may be different from each other, and thus aslight difference may occur between positions of an image projected onthe sensor unit 22 in the odd-numbered frame period and an imageprojected on the sensor unit 22 in the even-numbered frame period. Forexample, the image projected on the sensor unit 22 in the odd-numberedframe period may be projected by moving upward or downward by as much asone cell from the image projected on the sensor unit 22 in theeven-numbered frame period.

The signal processing unit 31 may output an image signal output from thesensor unit 22 corresponding to a frame period by converting the imagesignal into a frame.

For example, as in FIG. 13, the signal processing unit 31 may outputimage signals by converting the image signals into frames 5 a and 5 bevery frame periods T1 and T2 and synchronize with each of theodd-numbered frame period T1 and the even-numbered frame period T2 tooutput the odd-numbered frame 5 a and the even-numbered frame 5 b.

The signal processing unit 31 may also calculate depth informationcorresponding to each cell, i.e., each pixel, based on a phasedifference between light received through each cell of the sensor unit22 and light radiated by the first and second light sources 11 a and 11b. The calculated depth information is stored corresponding to eachpixel forming a frame.

The deinterlacer 32 may receive each of an odd-numbered frame and aneven-numbered frame from the signal processing unit 31 and merge the twoframes to generate one depth image.

As illustrated in FIG. 10, the deinterlacer 32 may generate a depthimage with a resolution that has increased twice compared to aresolution of the sensor unit 22 by alternately inserting anodd-numbered frame and an even-numbered frame output from the signalprocessing unit 31 in units of lines.

FIG. 14 is a flowchart illustrating a method of extracting depthinformation by the depth information extracting device according to yetanother embodiment of the present disclosure.

Referring to FIG. 14, the depth sensor module 10 controls the firstlight source 11 a to radiate light by synchronizing with an odd-numberedframe period (S201).

In Step S201, an irradiation region of the light radiated from the firstlight source 11 a may be controlled by the first holographic element 13a.

As the light radiated from the first light source 11 a is reflected froman object and incident on the sensor unit 22, the sensor unit 22generates an image signal corresponding to the light in units of pixels.In addition, the signal processing unit 31 performs a signal processingof the image signal output from the sensor unit 22 to generate anodd-numbered frame (S202).

Next, the depth sensor module 10 controls the second light source 11 bto radiate light by synchronizing with an even-numbered frame period(S203).

In Step S203, the light radiated from the second light source 11 b maybe radiated by moving upward or downward from the radiation region inStep S201 by the second holographic element 13 b.

As the light radiated from the second light source 11 b is reflectedfrom an object and incident on the sensor unit 22, the sensor unit 22generates an image signal corresponding to the light in units of pixels.In addition, the signal processing unit 31 performs a signal processingof the image signal output from the sensor unit 22 to generate aneven-numbered frame (S204).

As the light radiated from the light source 11 is reflected from anobject and incident on the sensor unit 22, the sensor unit 22 generatesan image signal corresponding to the light in units of pixels. Inaddition, the signal processing unit 31 performs a signal processing ofthe image signal output from the sensor unit 22 to generate aneven-numbered frame (S204).

The deinterlacer 32 generates a depth image by merging an odd-numberedframe and an even-numbered frame in units of lines of pixel arrays thatform the frames (S205).

In Step S106, the deinterlacer 32 may generate a depth image bysequentially and alternately merging lines of pixels of the odd-numberedframe and the even-numbered frame.

According to an embodiment of the present disclosure, one depth image isgenerated by adjusting irradiation regions of a light source to bedifferent between an odd-numbered frame period and an even-numberedframe period using a holographic element and merging an odd-numberedframe and an even-numbered frame obtained by the above method, therebyhaving an effect of improving resolution of the depth image.

In addition, a holographic element is used to adjust the emission angleof light, thereby having an effect of improving optical efficiency.

Although the present disclosure has been described with reference to theexemplary embodiments of the present disclosure, those of ordinary skillin the art should understand that the present disclosure may be modifiedand changed in various ways within the scope not departing from thespirit and area of the present disclosure described in the claims below.

1. A device for extracting depth information, the device comprising: alight outputting unit configured to output infrared (IR) light; a lightinputting unit configured to be input with light output from the lightoutputting unit and then reflected from an object; a light adjustingunit configured to adjust the angle of the light such that a firstregion including an object is irradiated with light and then adjust theangle of the light such a second region is irradiated with the light;and a controlling unit configured to estimate motion of the object usingat least one of light input into the first region and light input intothe second region.
 2. The device of claim 1, wherein the first region isan entire region including the object, and the second region isextracted from the first region and is a partial region including apredetermined region of the object.
 3. The device of claim 2, whereinthe controlling unit calculates a time of flight taken for light outputfrom the light outputting unit to be input into the light inputting unitafter being reflected from the second region that is irradiated with thelight.
 4. The device of claim 3, wherein the controlling unit estimatesmotion in the second region using a first time of flight calculatedusing light input at a first time point, a second time of flightcalculated using light input before the first time point, and a thirdtime of flight calculated using light input after a second time point.5. The device of claim 4, wherein the controlling unit estimates motionin the second region using relative differences between the first timeof flight, the second time of flight, and the third time of flight andthen compensates for the estimated motion using an interpolationtechnique.
 6. The device of claim 3, wherein: the light inputting unitincludes a plurality of pixels each including a first reception unit anda second reception unit; and the controlling unit calculates the time offlight using a difference in amounts of light input into the firstreception unit and the second reception unit.
 7. The device of claim 1,wherein the light adjusting unit includes a microelectromechanicalsystem (MEMS) and a MEMS controlling unit.
 8. The device of claim 1,wherein the controlling unit includes: a timing controlling unitconfigured to control time points of the light outputting unit, thelight adjusting unit, and the light inputting unit; a conversion unitconfigured to convert an electrical signal input through the lightinputting unit into a digital signal; and a signal processing unitconfigured to calculate times of flight of beams of light sequentiallyinput into the second region over time, estimate a motion in the secondregion, and extract depth information of the second region.
 9. Thedevice of claim 1, wherein the light adjusting unit includes aholographic element arranged between the light outputting unit and theobject and an actuator configured to drive the holographic element. 10.The device of claim 1, wherein: the first region and the second regionare regions that do not overlap each other; and the light adjusting unitadjusts the angle of light so that a region that is irradiated with thelight changes at every frame period.
 11. A method for extracting depthinformation of a device for extracting depth information, the methodcomprising: irradiating a first region including an object with infrared(IR) light; irradiating a second region with IR light; and estimatingmotion of the object using beams of light sequentially input into atleast one of the first region and the second region over time.
 12. Themethod of claim 11, wherein the estimating includes: calculating a firsttime of flight with respect to light input at a first time point, asecond time of flight with respect to light input before the first timepoint, and a third time of flight with respect to light input after asecond time point; and estimating motion in the second region usingrelative differences between the first time of flight, the second timeof flight, and the third time of flight.